The present invention is related to the compositing or blending of graphical image data.
Many graphics or image processing computer programs, such as Adobe(copyright) Photoshop(copyright), available from Adobe Systems Incorporated, of San Jose, Calif., build a final image by compositing two or more image layers together. The image layers may be thought of as stacked sheets of acetate. The density of the ink on the acetate controls the transparency of the sheet, i.e., the extent to which that sheet obscures the images from the underlying sheets. In the computer program, the color and density of the ink on the acetate sheet are represented by a color value and an alpha value (representing opacity), respectively. Normally, an alpha value of zero means the corresponding region is fully transparent, and a value of one means it is fully opaque.
Each layer typically includes image data and compositing controls. Typically, the image data is represented by an array of pixels, with each pixel having a color and, optionally, an opacity. Similarly, if there is a mask, the mask is represented by an array of pixels, with each pixel having an opacity. Alternatively, the image data or the mask or both can be defined analytically, e.g., by using shape outlines, or by other functions which map positions to color and opacity. In addition, the image data and the mask can be dynamic, that is, computed from other data at the time the layers are composited.
The compositing controls may include a transfer mode, also known as a blending mode. The transfer mode of an image layer determines how the color in the image layer mixes with the color accumulated from the underlying layers in the same image position. More specifically, it is a recipe for blending colors that can be expressed as a function taking an upper color and lower color to produce a third color. Image manipulation computer programs, such as Photoshop(copyright), generally provide a wide variety of predefined transfer modes. The basic transfer mode equation is:
new_color=xcex1xc2x7T(lower_color, upper_color)+(1xe2x88x92xcex1)xc2x7lower_color
where xcex1 is the opacity of the upper layer, T is the selected transfer mode function, and lower_color and upper_color are the color values of the lower layer (which in some contexts is called the accumulation layer) and the upper layer for the graphical element whose color new_color is being calculated. The color values are not pre-multiplied by the opacity. Transfer modes should not be confused with the well-known Porter-Duff compositing operations, which define how the alpha values determine how much of the blended colors survive in the result, but which assume essentially normal mode color mixing. T. Porter and T. Duff, xe2x80x9cCompositing Digital Imagesxe2x80x9d, SIGGRAPH 84, pp. 253-59 (1984).
The following table shows examples of transfer mode functions T. Each of the example transfer modes works componentwise; that is, each new colorant value is calculated based solely on the corresponding argument colorant values and independently of the values of different colorants. However, in the general case, transfer mode functions are not so limited. For example, there are hue, color, saturation, and luminosity modes that are based on doing a conversion to a hue, saturation, and luminance form (not exactly the same as either HSL or HSB because luminance is computed as a weighted average of red, green, and blue or cyan, magenta, and yellow multiplied by black), taking an appropriate mixture of the components from the upper and lower colors, and then performing a hue preserving clip to convert the color back into the RGB or CMY form. As is conventional, component colorants are treated as ranging from zero to one, with zero representing the darker end of the range, although in a typical implementation the actual values representing zero to one generally range across the values available at the colorants"" bit depth. The exact effect of the blending mode will depend on the color space. The convention that zero is darker than one for each colorant means that 100% ink in CMYK is represented by zero and 0% ink is represented by one, whereas in RGB, no phosphor excitation is represented by zero and full excitation is represented by one.
Image layers are generally composited in order from bottom to top. The general process for compositing an image layer begins with calculation of any dynamic data in the image layer, such as the color of the pixels in the image and the opacity of the pixels in the mask. Then, the opacity is determined for each pixel from the mask, the global opacity, if any, and the image data. Finally, the color of each pixel in the layer is combined with the color of the corresponding pixel in an accumulation buffer to generate a new composited color. The combination is controlled by the opacity of the pixel and the transfer mode.
Some computer programs, in addition to having predefined transfer modes, also give users the ability to define their own transfer modes. One such system is described in commonly-owned U.S. patent application Ser. No. 09/008,269 filed Jan. 16, 1998 to Hamburg, incorporated here by this reference, which discloses methods and apparatus, including computer programs, for use by a user to create new transfer modes (called compound transfer modes) based on existing (or newly-created) transfer modes and a scalar parameter, as appropriate.
In this specification, the terms color, alpha, and graphic element are used. A color is a representation of a particular color. The representation can be in any form suitable for computation and need only have the property that colors can be interpolated. Alpha is a quantity that characterizes the opacity of a graphic element. An alpha value of zero indicates total transparency. An alpha value of one indicates complete opacity. When working with graphic elements described with a color and alpha value, the color is free to be undefined or arbitrary if the alpha value is zero. A graphic element is a piece of the compositing process. Two graphic elements each having a color and alpha value are composited to produce a third graphic element. In a raster context, the primitive graphic element is a pixel. In a vector graphics context, it could be a region of solid color, such as a region defined by a PostScript path.
Raster accumulation is one example of compositing. In raster accumulation, the primitive graphic element is the pixel. The accumulation buffer is a collection of pixels arranged in a regular patternxe2x80x94generally a grid. The elements to be accumulated are also collections of pixels. Every pixel has a color and may have an alpha value. To composite a single element into the accumulation buffer, each pixel in the accumulation buffer is composited with the corresponding pixel in the element to produce a new value for the accumulation buffer pixel. A stack of elements is composited by initializing the accumulation buffer and then compositing each element from bottom to top into the accumulation buffer. The final contents of the accumulation buffer represent the final image.
At least one prior art system provided compositing with transfer mode T for color blending without alpha in an accumulation buffer (i.e., with an implicit alpha value of one) as follows.
where
InterpolateColor(A, B, 0)=A
and
xe2x80x83InterpolateColor(A, B, 1)=B.
Conventionally, color interpolation is linear, and defined as
InterpolateColor(A, B, xcex1)=Axc2x7(1xe2x88x92xcex1)+Bxc2x7xcex1
with each color component (colorant) value computed separately.
Similarly, a prior art system provided compositing without transfer modes but with accumulated alpha as follows.
The foregoing calculation was also used with a transfer mode that was the normal mode, that is, where T(lower, upper)=upper.
Where compositing involved both a transfer mode and an accumulated alpha, the prior art generally provided ad hoc solutions designed to work with specific transfer modes or kinds of transfer modes. Such ad hoc solutions involved doing things like temporarily matting the results in the accumulation buffer against a solid color or extending the accumulation buffer with special values. For example, a prior art technique for dealing with multiply mode is to extend the lower color and opacity with white before blending in the upper color. As a consequence, where there is no lower color, the result of the blend is the upper color. This technique works if something can play the role of white, but the technique cannot be applied in general. Another technique of ad hoc prior art systems is simply to expect pixel colors to be defined even where the alpha value is zero. In effect, this initializes the accumulation buffer to white or some other color wherever alpha is zero.
Another ad hoc method does a matte with a color, blend, and dematte. This method is applicable where a color M exists such that T(M,X)=X, in which case the following calculations are performed. In the calculations, A is the accumulated_color, a is the accumulated_alpha, B is the element_color, and b is the element_alpha.
A:=Mxc2x7(1xe2x88x92a)+Axc2x7a
A:=Axc2x7(1xe2x88x92b)+T(A,B)xc2x7b
a:=a+bxe2x88x92axc2x7b
A:=(Axe2x88x92Mxc2x7(1xe2x88x92a))/a
Yet another ad hoc method extends with element data and then blend. To work properly, this method depends on the transfer mode satisfying T(X,X)=X. In terms of the definitions used above, the following calculations are performed by this method.
x:=a
a:=a+bxe2x88x92axc2x7b
t:=x/a
A:=Bxc2x7(1xe2x88x92t)+Axc2x7t
A:=Axc2x7(1xe2x88x92b)+T(A,B)xc2x7b
None of the ad hoc methods provides satisfactory results for ColorDodge or ColorBurn transfer modes, and none is suitable for use with arbitrary, user-defined transfer modes.
The invention features methods and apparatus, including computer program products, operable to blend two color-and-alpha pairs using any arbitrary transfer mode function to specify the color blending, providing, in the general case, transparency information in the result. The invention is particularly useful for accumulating a color-and-alpha value into an accumulation buffer, but it is generally useful for producing a blended color and alpha value given two color-and-alpha pairs and an arbitrary transfer mode.
In general, in one aspect, the invention includes blending a lower and an upper graphic element in accordance with an arbitrary transfer mode by calculating a transfer color from a lower element color and an upper element color according to the transfer mode without regard either to a lower element alpha or to an upper element alpha; calculating a resulting alpha from the lower element alpha and the upper element alpha without regard to any of the lower element color, the upper element color, or the transfer mode; and calculating a resulting color as a weighted average of the lower element color, the transfer color, and the upper element color. Advantageous implementations can include one or more of the following features. The weighted average is calculated as an interpolation of the lower element color and the upper element color interpolated with the transfer color, where the interpolation of the upper element color is governed by the lower element alpha and the interpolation of the lower element color is governed by the upper element alpha divided by the resulting alpha. The transfer mode can be user defined. Each of the lower and upper elements can have an alpha value between fully transparent and fully opaque, exclusive. Each of the lower and upper elements can be a pixel or a region. Each of the colors can be defined by a value for each of a plurality of colorants.
In general, in another aspect, the invention can be applied to accumulating graphic layers in a layer order. In this aspect, the invention includes providing an accumulation buffer having a plurality of graphic elements, each graphic element having color and alpha values; compositing the graphic elements of the layers into the accumulation buffer in the layer order; for each compositing of an accumulation element and a layer element of a layer having an associated transfer mode function, (i) calculating a resulting alpha value from the alpha values of the accumulation element and the layer element, (ii) calculating a resulting color value from a color value of the accumulation element, a color value of the layer element, and the transfer mode function, and (iii) updating the accumulation buffer by storing the resulting color value and the resulting alpha value in the accumulation buffer; wherein the resulting color is a weighted average of the color value of the accumulation element, the color value of the layer element, and the value of the transfer mode function applied to the color value of the accumulation element and the color value of the layer element, the weights for the weighted average being determined by the alpha value of the accumulation element and the alpha value of the layer element. The invention is advantageously applied in situations where, after accumulating at least one of the layers into the accumulation buffer, not all the elements of the accumulation buffer have the same alpha value; where at least one of the resulting alpha values stored in the accumulation buffer is an alpha value between fully transparent and fully opaque, exclusive; where at least one of the elements of at least one of the second through last layers in the layer order has an alpha value between fully transparent and fully opaque, exclusive; where the transfer mode associated with at least one of the layers lightens the accumulation color; or where the transfer mode associated with at least one of the layers applies different lightening or darkening effects depending on the color of the layer element.
Advantages that can be seen in implementations of the invention include one or more of the following. The invention provides fully general handling of transfer modes in compositing with accumulated alpha and does so in a consistent manner producing sensible, visually continuous and therefore pleasing results, particularly when compositing into an accumulation buffer having both zero and non-zero alpha values. The compositing results match the two special cases described abovexe2x80x94compositing without transfer modes but with accumulated alpha and compositing without accumulated alpha but with transfer modesxe2x80x94and do at least as well as current special-case-based solutions. In particular, for a source color, a source alpha value, and a transfer mode being blended into an accumulation buffer, the following desirable conditions are satisfied.
(a) If the accumulation buffer is entirely transparentxe2x80x94i.e., the accumulation alpha value is 0xe2x80x94there is nothing to blend with, and the resulting accumulated color and alpha value is the same as source color and source alpha.
(b) If the accumulation buffer is fully opaquexe2x80x94i.e., the alpha value is 1xe2x80x94the resulting color is the same as in the case where the accumulation buffer simply does not have an alpha component and the accumulated alpha value remains 1.
(c) The intermediate results are visually continuousxe2x80x94i.e., if the initial accumulation buffer contains an alpha value of 1%, then the results are very similar to the case where it contains an alpha value of 0; if the accumulation buffer contains an alpha value of 99%, then the results are very similar to the 100% case.
(d) If the source alpha value is zero, the resulting accumulated color and alpha value match the initial accumulated color and alpha value.
(e) The intermediate results are visually continuous with changes in the source opacity.
(f) Alpha values are combined in the same way with and without a transfer mode.
(g) The compositing operation does not depend on special properties of a selected transfer mode and therefore operates easily with new transfer mode functions, including user-defined functions.
In addition, the invention simplifies the design of graphics manipulation systems, whether implemented in hardware, software, firmware, or any combination of them. A system using the invention can accept as input any number of layers of graphic elements having any arbitrary selection of color and alpha values, and can further accept and associate with any of the layers any arbitrary transfer mode function, and composite the elements according to the transfer modes in a fully general way. The invention allows the colors for the accumulation or the source to be arbitrary when the accumulated alpha or source alpha is zero, respectively.
The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features and advantages of the invention will become apparent from the description, the drawings, and the claims.